{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eaed3927-e315-46d3-8889-df3f3bbcbf6b",
   "metadata": {},
   "source": [
    "# Quantize a Hugging Face Question-Answering Model with OpenVINO\n",
    "\n",
    "This notebook shows how to quantize a question answering model with OpenVINO's [Neural Network Compression Framework](https://github.com/openvinotoolkit/nncf) (NNCF). \n",
    "\n",
    "With quantization, we reduce the precision of the model's weights and activations from floating point (FP32) to integer (INT8). This results in a smaller model with faster inference times with OpenVINO Runtime. \n",
    "\n",
    "The notebook demonstrates post-training quantization, which does not require specific hardware to execute. A laptop or desktop with a recent Intel Core processor is recommended for best results. To install the requirements for this notebook, please do `pip install -r requirements.txt` or uncomment the cell below to install the requirements in your current Python environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dffab375-a730-4015-8d17-360b76a0718d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install \"optimum-intel[openvino,nncf]\" datasets evaluate[evaluator] ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0407fc92-c052-47b7-8721-01836adf3b54",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:23.169301Z",
     "iopub.status.busy": "2022-12-04T14:40:23.169096Z",
     "iopub.status.idle": "2022-12-04T14:40:25.146758Z",
     "shell.execute_reply": "2022-12-04T14:40:25.146281Z",
     "shell.execute_reply.started": "2022-12-04T14:40:23.169248Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import warnings\n",
    "from pathlib import Path\n",
    "\n",
    "import datasets\n",
    "import evaluate\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import transformers\n",
    "from evaluate import evaluator\n",
    "from openvino import Core\n",
    "from transformers import AutoModelForQuestionAnswering, AutoTokenizer, pipeline\n",
    "\n",
    "from optimum.intel import OVConfig, OVModelForQuestionAnswering, OVQuantizationConfig, OVQuantizer\n",
    "\n",
    "\n",
    "transformers.logging.set_verbosity_error()\n",
    "datasets.logging.set_verbosity_error()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "889a16fe-2bc0-477e-b8d6-02a4f7508f03",
   "metadata": {},
   "source": [
    "## Settings\n",
    "\n",
    "We define MODEL_ID and DATASET_NAME, and the paths for the quantized model files. VERSION_2_WITH_NEGATIVE should be set to TRUE if a version of the SQuAD v2 dataset is used, which includes questions that do not have an answer. \n",
    "\n",
    "For this tutorial, we use the [Stanford Question Answering Dataset (SQuAD)](https://huggingface.co/datasets/squad), a reading comprehension dataset consisting of questions on a set of Wikipedia articles, where the answer to every question is a segment of text from a given context. The notebook was tested with the [csarron/bert-base-uncased-squad-v1](https://huggingface.co/csarron/bert-base-uncased-squad-v1) model. Other [question-answering models](https://huggingface.co/models?dataset=dataset:squad&pipeline_tag=question-answering&sort=downloads) should also work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c32f9a76-414b-43d9-9769-af131223f1c1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:25.147946Z",
     "iopub.status.busy": "2022-12-04T14:40:25.147671Z",
     "iopub.status.idle": "2022-12-04T14:40:25.150427Z",
     "shell.execute_reply": "2022-12-04T14:40:25.150085Z",
     "shell.execute_reply.started": "2022-12-04T14:40:25.147933Z"
    }
   },
   "outputs": [],
   "source": [
    "MODEL_ID = \"csarron/bert-base-uncased-squad-v1\"\n",
    "DATASET_NAME = \"squad\"\n",
    "VERSION_2_WITH_NEGATIVE = False\n",
    "\n",
    "base_model_path = Path(f\"models/{MODEL_ID}\")\n",
    "fp32_model_path = base_model_path.with_name(base_model_path.name + \"_FP32\")\n",
    "int8_ptq_model_path = base_model_path.with_name(base_model_path.name + \"_INT8_PTQ\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c3547df-5603-41b4-8752-df5fb7347be0",
   "metadata": {},
   "source": [
    "## Load Model and Tokenizer\n",
    "\n",
    "We load the model from the Hugging Face Hub. The model will be automatically downloaded if it has not been downloaded before, or loaded from the cache otherwise.\n",
    "\n",
    "We also load the tokenizer, which converts the questions and contexts from the dataset to tokens, converting the inputs in a format the model expects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "38641b14-07d0-49d5-af86-8b5247ae39d8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:25.151098Z",
     "iopub.status.busy": "2022-12-04T14:40:25.150929Z",
     "iopub.status.idle": "2022-12-04T14:40:27.655923Z",
     "shell.execute_reply": "2022-12-04T14:40:27.655188Z",
     "shell.execute_reply.started": "2022-12-04T14:40:25.151086Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input_ids': [101, 7592, 2088, 999, 102], 'token_type_ids': [0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1]}\n"
     ]
    }
   ],
   "source": [
    "model = AutoModelForQuestionAnswering.from_pretrained(MODEL_ID)\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)\n",
    "\n",
    "# See how the tokenizer for the given model converts input text to model input values\n",
    "print(tokenizer(\"hello world!\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "124bd9ad-077c-4f41-b579-0bf978fe6a1e",
   "metadata": {},
   "source": [
    "## Preview the Dataset\n",
    "\n",
    "The `datasets` library makes it easy to load datasets. Common datasets can be loaded from the Hugging Face Hub by providing the name of the dataset. See https://github.com/huggingface/datasets. We can load the SQuAD dataset with `load_dataset` and show a random dataset item. Every dataset item in the SQuAD dataset has a unique id, a title which denotes the category, a context and a question, and answers. The answer is a subset of the context, and both the text of the answer, and the start position of the answer in the context (`answer_start`) are returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "602fe46f-c96a-4a0f-9338-58339d466f3a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:27.656861Z",
     "iopub.status.busy": "2022-12-04T14:40:27.656663Z",
     "iopub.status.idle": "2022-12-04T14:40:31.357513Z",
     "shell.execute_reply": "2022-12-04T14:40:31.357057Z",
     "shell.execute_reply.started": "2022-12-04T14:40:27.656848Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "80b731d8fc214bf086474014387ab6fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'id': '570e53690b85d914000d7e3c',\n",
       " 'title': 'Melbourne',\n",
       " 'context': \"Melbourne is experiencing high population growth, generating high demand for housing. This housing boom has increased house prices and rents, as well as the availability of all types of housing. Subdivision regularly occurs in the outer areas of Melbourne, with numerous developers offering house and land packages. However, after 10 years[when?] of planning policies to encourage medium-density and high-density development in existing areas with greater access to public transport and other services, Melbourne's middle and outer-ring suburbs have seen significant brownfields redevelopment.\",\n",
       " 'question': 'What effect has the housing boom had on house prices and rents?',\n",
       " 'answers': {'text': ['increased'], 'answer_start': [108]}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.load_dataset(DATASET_NAME)\n",
    "dataset[\"train\"][31415]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce4dfc2d-f007-4455-8043-edc5468b87e2",
   "metadata": {},
   "source": [
    "## Post Training Quantization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5483309f-0b12-4c10-ab0b-58c50981f495",
   "metadata": {
    "tags": []
   },
   "source": [
    "For post-training quantization (PTQ), we first start by loading the model using the `AutoModelForQuestionAnswering` class. After instantiating an `OVQuantizer`, we need to provide a dataset for the calibration step. You can apply quantization on your model by calling the `quantize` method. That's all!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b14899a-3aec-46b6-9d4b-332452e9cb25",
   "metadata": {},
   "source": [
    "### Prepare the Dataset\n",
    "\n",
    "We need a representative calibration dataset to quantize the model. The SQuAD dataset is pretrained on a large dataset with a wide variety of questions and answers, and it generalizes pretty well to questions and contexts it has never seen before. For production use, you would finetune this dataset with questions and context specific to your domain. In this notebook, we use a subset of the SQuAD dataset, for demonstration purposes. We chose the _Super Bowl 50_ category from the validation subset of SQuAD because it has a large number of questions.\n",
    "\n",
    "Post-training quantization does not need a training and validation dataset, because we will not train the model, but we define these splits here to use a training split for calibration, and a validation split for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8be68958-ce5e-4cc6-b8e7-2867feaf084b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:31.358230Z",
     "iopub.status.busy": "2022-12-04T14:40:31.358053Z",
     "iopub.status.idle": "2022-12-04T14:40:31.360666Z",
     "shell.execute_reply": "2022-12-04T14:40:31.360301Z",
     "shell.execute_reply.started": "2022-12-04T14:40:31.358218Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def preprocess_fn(examples, tokenizer):\n",
    "    \"\"\"convert the text from the dataset into tokens in the format that the model expects\"\"\"\n",
    "    return tokenizer(\n",
    "        examples[\"question\"],\n",
    "        examples[\"context\"],\n",
    "        padding=True,\n",
    "        truncation=True,\n",
    "        max_length=384,\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5b4dde1a-1ba6-470c-afe8-7169c04282dc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:31.361367Z",
     "iopub.status.busy": "2022-12-04T14:40:31.361175Z",
     "iopub.status.idle": "2022-12-04T14:40:31.401437Z",
     "shell.execute_reply": "2022-12-04T14:40:31.401053Z",
     "shell.execute_reply.started": "2022-12-04T14:40:31.361351Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "NUM_TRAIN_ITEMS = 600\n",
    "filtered_examples = dataset[\"validation\"].filter(lambda x: x[\"title\"].startswith(\"Super_Bowl_50\"))\n",
    "train_examples = filtered_examples.select(range(0, NUM_TRAIN_ITEMS))\n",
    "train_dataset = train_examples.map(lambda x: preprocess_fn(x, tokenizer), batched=True)\n",
    "\n",
    "validation_examples = filtered_examples.select(range(NUM_TRAIN_ITEMS, len(filtered_examples)))\n",
    "validation_dataset = validation_examples.map(lambda x: preprocess_fn(x, tokenizer), batched=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb0b0738-fdc9-4557-97bf-b4c6709280cc",
   "metadata": {},
   "source": [
    "### Quantize the Model with Post Training Quantization\n",
    "\n",
    "**NOTE:** if you notice very low accuracy after post-training quantization, it is likely caused by an overflow issue which affects processors that do not contain VNNI (Vector Neural Network Instruction). NNCF has an `overflow_fix` option to address this. It will effectively use 7-bits for quantizing instead of 8-bits to prevent the overflow. To use this option, modify the code in the next cell to add an explicit quantization configuration, and set `overflow_fix` to `\"enable\"`:\n",
    "\n",
    "```\n",
    "from optimum.intel import OVConfig, OVQuantizationConfig\n",
    "\n",
    "ov_config = OVConfig(quantization_config=OVQuantizationConfig(overflow_fix=\"enable\")\n",
    "quantizer = OVQuantizer.from_pretrained(model)\n",
    "quantizer.quantize(calibration_dataset=train_dataset, save_directory=int8_ptq_model_path, ov_config=ov_config)\n",
    "```\n",
    "\n",
    "For more information, see [Lower Numerical Precision Deep Learning Inference and Training](https://www.intel.com/content/www/us/en/developer/articles/technical/lower-numerical-precision-deep-learning-inference-and-training.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "31c5415e-e22b-4ab9-b903-8791e80b188d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:40:31.402637Z",
     "iopub.status.busy": "2022-12-04T14:40:31.402463Z",
     "iopub.status.idle": "2022-12-04T14:42:55.379929Z",
     "shell.execute_reply": "2022-12-04T14:42:55.379439Z",
     "shell.execute_reply.started": "2022-12-04T14:40:31.402625Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Hide PyTorch warnings about missing shape inference\n",
    "warnings.simplefilter(\"ignore\")\n",
    "\n",
    "# Quantize the model\n",
    "quantizer = OVQuantizer.from_pretrained(model)\n",
    "ov_config = OVConfig(quantization_config=OVQuantizationConfig())\n",
    "quantizer.quantize(calibration_dataset=train_dataset, ov_config=ov_config, save_directory=int8_ptq_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2574cc63-aad3-4c28-aa6f-e553de911ce5",
   "metadata": {},
   "source": [
    "## Compare INT8 and FP32 models\n",
    "\n",
    "We compare the accuracy, model size and inference results and latency of the FP32 and INT8 models.\n",
    "### Inference Pipeline\n",
    "\n",
    "Transformers [Pipelines](https://huggingface.co/docs/transformers/main/en/pipeline_tutorial) simplify model inference. A `Pipeline` is created by adding a task, model and tokenizer to the `pipeline` function. Inference is then as simple as `qa_pipeline({\"question\": question, \"context\": context})`.\n",
    "\n",
    "We create two pipelines: `hf_qa_pipeline` and `ov_qa_pipeline_ptq` to compare the FP32 PyTorch model with the OpenVINO INT8 model. These pipelines will also be used for showing the accuracy difference and for benchmarking later in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "28ddc37f-5504-4eb1-91c7-063d6f1a8174",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:42:55.380868Z",
     "iopub.status.busy": "2022-12-04T14:42:55.380738Z",
     "iopub.status.idle": "2022-12-04T14:42:58.064848Z",
     "shell.execute_reply": "2022-12-04T14:42:58.064382Z",
     "shell.execute_reply.started": "2022-12-04T14:42:55.380857Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "quantized_model_ptq = OVModelForQuestionAnswering.from_pretrained(int8_ptq_model_path)\n",
    "original_model = AutoModelForQuestionAnswering.from_pretrained(MODEL_ID)\n",
    "ov_qa_pipeline_ptq = pipeline(\"question-answering\", model=quantized_model_ptq, tokenizer=tokenizer)\n",
    "hf_qa_pipeline = pipeline(\"question-answering\", model=original_model, tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2e23fe96-8d7f-4aa1-816f-707ca1a2f978",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:42:58.065629Z",
     "iopub.status.busy": "2022-12-04T14:42:58.065437Z",
     "iopub.status.idle": "2022-12-04T14:42:58.068482Z",
     "shell.execute_reply": "2022-12-04T14:42:58.068047Z",
     "shell.execute_reply.started": "2022-12-04T14:42:58.065617Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Super Bowl 50 featured numerous records from individuals and teams. Denver won despite being massively outgained in total yards (315 to 194) and first downs (21 to 11). Their 194 yards and 11 first downs were both the lowest totals ever by a Super Bowl winning team. The previous record was 244 yards by the Baltimore Ravens in Super Bowl XXXV. Only seven other teams had ever gained less than 200 yards in a Super Bowl, and all of them had lost. The Broncos' seven sacks tied a Super Bowl record set by the Chicago Bears in Super Bowl XX. Kony Ealy tied a Super Bowl record with three sacks. Jordan Norwood's 61-yard punt return set a new record, surpassing the old record of 45 yards set by John Taylor in Super Bowl XXIII. Denver was just 1-of-14 on third down, while Carolina was barely better at 3-of-15. The two teams' combined third down conversion percentage of 13.8 was a Super Bowl low. Manning and Newton had quarterback passer ratings of 56.6 and 55.4, respectively, and their added total of 112 is a record lowest aggregate passer rating for a Super Bowl. Manning became the oldest quarterback ever to win a Super Bowl at age 39, and the first quarterback ever to win a Super Bowl with two different teams, while Gary Kubiak became the first head coach to win a Super Bowl with the same franchise he went to the Super Bowl with as a player.\n"
     ]
    }
   ],
   "source": [
    "context = validation_examples[200][\"context\"]\n",
    "question = \"Who won the game?\"\n",
    "print(context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c1168f1c-14de-4aad-977d-122a8d366935",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:42:58.069207Z",
     "iopub.status.busy": "2022-12-04T14:42:58.068940Z",
     "iopub.status.idle": "2022-12-04T14:42:58.335534Z",
     "shell.execute_reply": "2022-12-04T14:42:58.334771Z",
     "shell.execute_reply.started": "2022-12-04T14:42:58.069194Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Denver'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hf_qa_pipeline({\"question\": question, \"context\": context})[\"answer\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c885d378-2842-49d0-b583-a2fc023558b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:42:58.336807Z",
     "iopub.status.busy": "2022-12-04T14:42:58.336478Z",
     "iopub.status.idle": "2022-12-04T14:42:58.495631Z",
     "shell.execute_reply": "2022-12-04T14:42:58.494774Z",
     "shell.execute_reply.started": "2022-12-04T14:42:58.336783Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Denver'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ov_qa_pipeline_ptq({\"question\": question, \"context\": context})[\"answer\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97a52092-e352-47ef-9ed2-89508bc48d70",
   "metadata": {},
   "source": [
    "### Accuracy\n",
    "\n",
    "We load the quantized model and the original FP32 model, and compare the metrics on both models. The [evaluate](https://github.com/huggingface/evaluate) library makes it very easy to evaluate models on a given dataset, with a given metric. For the SQuAD dataset, the F1 score and Exact Match metrics are returned.\n",
    "\n",
    "To load the quantized model with OpenVINO, we use the `OVModelForQuestionAnswering` class. It can be used in the same way as [`AutoModelForQuestionAnswering`](https://huggingface.co/docs/transformers/main/model_doc/auto).\n",
    "\n",
    "The pipelines we created in the previous section are used to perform evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6c2f615a-19e3-4ee2-9309-2ae1392c7f62",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:42:58.497409Z",
     "iopub.status.busy": "2022-12-04T14:42:58.496740Z",
     "iopub.status.idle": "2022-12-04T14:43:47.884825Z",
     "shell.execute_reply": "2022-12-04T14:43:47.884058Z",
     "shell.execute_reply.started": "2022-12-04T14:42:58.497370Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>exact_match</th>\n",
       "      <th>f1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>FP32</th>\n",
       "      <td>82.86</td>\n",
       "      <td>86.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>INT8 PTQ</th>\n",
       "      <td>82.86</td>\n",
       "      <td>87.42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          exact_match     f1\n",
       "FP32            82.86  86.33\n",
       "INT8 PTQ        82.86  87.42"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "squad_eval = evaluator(\"question-answering\")\n",
    "\n",
    "ov_eval_results = squad_eval.compute(\n",
    "    model_or_pipeline=ov_qa_pipeline_ptq,\n",
    "    data=validation_examples,\n",
    "    metric=\"squad\",\n",
    "    squad_v2_format=VERSION_2_WITH_NEGATIVE,\n",
    ")\n",
    "\n",
    "hf_eval_results = squad_eval.compute(\n",
    "    model_or_pipeline=hf_qa_pipeline,\n",
    "    data=validation_examples,\n",
    "    metric=\"squad\",\n",
    "    squad_v2_format=VERSION_2_WITH_NEGATIVE,\n",
    ")\n",
    "pd.DataFrame.from_records(\n",
    "    [hf_eval_results, ov_eval_results],\n",
    "    columns=[\"exact_match\", \"f1\"],\n",
    "    index=[\"FP32\", \"INT8 PTQ\"],\n",
    ").round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db183795-6dae-4ef6-847d-042223264149",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-11-07T21:25:39.912874Z",
     "iopub.status.busy": "2022-11-07T21:25:39.912662Z",
     "iopub.status.idle": "2022-11-07T21:25:39.916029Z",
     "shell.execute_reply": "2022-11-07T21:25:39.915541Z",
     "shell.execute_reply.started": "2022-11-07T21:25:39.912859Z"
    }
   },
   "source": [
    "### Inference Results\n",
    "\n",
    "To fully understand the quality of a model, it is useful to look beyond metrics like Exact Match and F1 score and examine model predictions directly. This can give a more complete impression of the model's performance and help identify areas for improvement.\n",
    "\n",
    "In the next cell, we go over the items in the validation set, and display the items where the FP32 prediction score is different from the INT8 prediction score\n",
    "\n",
    "The results show that for some predictions, the FP32 model is better, but for others, the INT8 model is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "256e4b8c-a791-4668-95e3-4d23720fcf94",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:43:47.886011Z",
     "iopub.status.busy": "2022-12-04T14:43:47.885656Z",
     "iopub.status.idle": "2022-12-04T14:44:36.260004Z",
     "shell.execute_reply": "2022-12-04T14:44:36.259360Z",
     "shell.execute_reply.started": "2022-12-04T14:43:47.885988Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Answer</th>\n",
       "      <th>FP32 answer</th>\n",
       "      <th>FP32 F1</th>\n",
       "      <th>INT8 answer</th>\n",
       "      <th>INT8 F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What company paid for a Super Bowl 50 ad to show a trailer of X-Men: Apocalypse?</td>\n",
       "      <td>[Fox, Fox, Disney]</td>\n",
       "      <td>20th Century Fox, Lionsgate</td>\n",
       "      <td>40.0</td>\n",
       "      <td>20th Century Fox, Lionsgate, Paramount Pictures, Universal Studios</td>\n",
       "      <td>22.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What BBC radio station will carry the game in the United Kingdom?</td>\n",
       "      <td>[BBC Radio 5, Radio 5 Live, BBC Radio 5 Live]</td>\n",
       "      <td>BBC Radio 5 Live</td>\n",
       "      <td>100.0</td>\n",
       "      <td>BBC Radio 5 Live and 5 Live Sports Extra</td>\n",
       "      <td>61.54</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Aside from BBC Radio 5, what radio station will broadcast the game?</td>\n",
       "      <td>[5 Live Sports Extra, 5 Live Sports Extra, 5 Live Sports Extra]</td>\n",
       "      <td>BBC Radio 5 Live</td>\n",
       "      <td>50.0</td>\n",
       "      <td>5 Live Sports Extra</td>\n",
       "      <td>100.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How many players have been awarded the Most Valuable Player distinction for the Super Bowl?</td>\n",
       "      <td>[43, 43, 43]</td>\n",
       "      <td>43</td>\n",
       "      <td>100.0</td>\n",
       "      <td>39 of the 43</td>\n",
       "      <td>50.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>How many former MVP honorees were present for a pregame ceremony?</td>\n",
       "      <td>[39, 39, 39]</td>\n",
       "      <td>43</td>\n",
       "      <td>0.0</td>\n",
       "      <td>39 of the 43</td>\n",
       "      <td>50.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How many yards was the missed field goal?</td>\n",
       "      <td>[44, 44, 44]</td>\n",
       "      <td>33</td>\n",
       "      <td>0.0</td>\n",
       "      <td>44</td>\n",
       "      <td>100.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Who picked off Cam Newton and subsequently fumbled the ball?</td>\n",
       "      <td>[T. J. Ward, T. J. Ward, Ward]</td>\n",
       "      <td>Trevathan</td>\n",
       "      <td>0.0</td>\n",
       "      <td>T. J. Ward</td>\n",
       "      <td>100.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>What yard line was the Broncos on when Manning lost the ball in the fourth quarter?</td>\n",
       "      <td>[50-yard line., 41, 50]</td>\n",
       "      <td>41</td>\n",
       "      <td>100.0</td>\n",
       "      <td>41-yard line</td>\n",
       "      <td>50.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Which player was criticized for not jumping into the pile to recover the ball?</td>\n",
       "      <td>[Newton, Newton, Newton]</td>\n",
       "      <td>Ward</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Newton</td>\n",
       "      <td>100.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>How many plays was Denver kept out of the end zone after getting the ball from Newton?</td>\n",
       "      <td>[three, three, three]</td>\n",
       "      <td>three</td>\n",
       "      <td>100.0</td>\n",
       "      <td>three plays</td>\n",
       "      <td>66.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>How many picks did Cam Newton throw?</td>\n",
       "      <td>[one, one, one]</td>\n",
       "      <td>18 of 41 passes for 265 yards</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18 of 41 passes for 265 yards, with one interception</td>\n",
       "      <td>18.18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                       Question  \\\n",
       "0              What company paid for a Super Bowl 50 ad to show a trailer of X-Men: Apocalypse?   \n",
       "1                             What BBC radio station will carry the game in the United Kingdom?   \n",
       "2                           Aside from BBC Radio 5, what radio station will broadcast the game?   \n",
       "3   How many players have been awarded the Most Valuable Player distinction for the Super Bowl?   \n",
       "4                             How many former MVP honorees were present for a pregame ceremony?   \n",
       "5                                                     How many yards was the missed field goal?   \n",
       "6                                  Who picked off Cam Newton and subsequently fumbled the ball?   \n",
       "7           What yard line was the Broncos on when Manning lost the ball in the fourth quarter?   \n",
       "8                Which player was criticized for not jumping into the pile to recover the ball?   \n",
       "9        How many plays was Denver kept out of the end zone after getting the ball from Newton?   \n",
       "10                                                         How many picks did Cam Newton throw?   \n",
       "\n",
       "                                                             Answer  \\\n",
       "0                                                [Fox, Fox, Disney]   \n",
       "1                     [BBC Radio 5, Radio 5 Live, BBC Radio 5 Live]   \n",
       "2   [5 Live Sports Extra, 5 Live Sports Extra, 5 Live Sports Extra]   \n",
       "3                                                      [43, 43, 43]   \n",
       "4                                                      [39, 39, 39]   \n",
       "5                                                      [44, 44, 44]   \n",
       "6                                    [T. J. Ward, T. J. Ward, Ward]   \n",
       "7                                           [50-yard line., 41, 50]   \n",
       "8                                          [Newton, Newton, Newton]   \n",
       "9                                             [three, three, three]   \n",
       "10                                                  [one, one, one]   \n",
       "\n",
       "                      FP32 answer  FP32 F1  \\\n",
       "0     20th Century Fox, Lionsgate     40.0   \n",
       "1                BBC Radio 5 Live    100.0   \n",
       "2                BBC Radio 5 Live     50.0   \n",
       "3                              43    100.0   \n",
       "4                              43      0.0   \n",
       "5                              33      0.0   \n",
       "6                       Trevathan      0.0   \n",
       "7                              41    100.0   \n",
       "8                            Ward      0.0   \n",
       "9                           three    100.0   \n",
       "10  18 of 41 passes for 265 yards      0.0   \n",
       "\n",
       "                                                           INT8 answer  \\\n",
       "0   20th Century Fox, Lionsgate, Paramount Pictures, Universal Studios   \n",
       "1                             BBC Radio 5 Live and 5 Live Sports Extra   \n",
       "2                                                  5 Live Sports Extra   \n",
       "3                                                         39 of the 43   \n",
       "4                                                         39 of the 43   \n",
       "5                                                                   44   \n",
       "6                                                           T. J. Ward   \n",
       "7                                                         41-yard line   \n",
       "8                                                               Newton   \n",
       "9                                                          three plays   \n",
       "10                18 of 41 passes for 265 yards, with one interception   \n",
       "\n",
       "    INT8 F1  \n",
       "0     22.22  \n",
       "1     61.54  \n",
       "2    100.00  \n",
       "3     50.00  \n",
       "4     50.00  \n",
       "5    100.00  \n",
       "6    100.00  \n",
       "7     50.00  \n",
       "8    100.00  \n",
       "9     66.67  \n",
       "10    18.18  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = []\n",
    "metric = evaluate.load(\"squad_v2\" if VERSION_2_WITH_NEGATIVE else \"squad\")\n",
    "\n",
    "for item in validation_examples:\n",
    "    id, title, context, question, answers = item.values()\n",
    "    fp32_answer = hf_qa_pipeline(question, context)[\"answer\"]\n",
    "    int8_answer = ov_qa_pipeline_ptq(question, context)[\"answer\"]\n",
    "\n",
    "    references = [{\"id\": id, \"answers\": answers}]\n",
    "    fp32_predictions = [{\"id\": id, \"prediction_text\": fp32_answer}]\n",
    "    int8_predictions = [{\"id\": id, \"prediction_text\": int8_answer}]\n",
    "\n",
    "    fp32_score = round(metric.compute(references=references, predictions=fp32_predictions)[\"f1\"], 2)\n",
    "    int8_score = round(metric.compute(references=references, predictions=int8_predictions)[\"f1\"], 2)\n",
    "\n",
    "    if int8_score != fp32_score:\n",
    "        results.append((question, answers[\"text\"], fp32_answer, fp32_score, int8_answer, int8_score))\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "pd.DataFrame(\n",
    "    results,\n",
    "    columns=[\"Question\", \"Answer\", \"FP32 answer\", \"FP32 F1\", \"INT8 answer\", \"INT8 F1\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58df445d-af43-4ba1-8195-7d8f00b8f82f",
   "metadata": {},
   "source": [
    "### Model Size\n",
    "\n",
    "We save the FP32 PyTorch model and define a function to show the model size for the PyTorch and OpenVINO models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1eeaa81f-7fc5-49ba-80b8-2d95a1310a0c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:44:36.261080Z",
     "iopub.status.busy": "2022-12-04T14:44:36.260880Z",
     "iopub.status.idle": "2022-12-04T14:44:37.575213Z",
     "shell.execute_reply": "2022-12-04T14:44:37.574836Z",
     "shell.execute_reply.started": "2022-12-04T14:44:36.261063Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FP32 model size: 436.50 MB\n",
      "INT8 model size: 181.84 MB\n",
      "INT8 size decrease: 2.4x\n"
     ]
    }
   ],
   "source": [
    "def get_model_size(model_folder, framework):\n",
    "    \"\"\"\n",
    "    Return OpenVINO or PyTorch model size in Mb.\n",
    "    Arguments:\n",
    "        model_folder:\n",
    "            Directory containing a model.safetensors for a PyTorch model, and an openvino_model.xml/.bin for an OpenVINO model.\n",
    "        framework:\n",
    "            Define whether the model is a PyTorch or an OpenVINO model.\n",
    "    \"\"\"\n",
    "    if framework.lower() == \"openvino\":\n",
    "        model_path = Path(model_folder) / \"openvino_model.xml\"\n",
    "        model_size = model_path.stat().st_size + model_path.with_suffix(\".bin\").stat().st_size\n",
    "    elif framework.lower() == \"pytorch\":\n",
    "        model_path = Path(model_folder) / \"model.safetensors\"\n",
    "        model_size = model_path.stat().st_size\n",
    "    model_size /= 1000 * 1000\n",
    "    return model_size\n",
    "\n",
    "\n",
    "model.save_pretrained(fp32_model_path)\n",
    "\n",
    "fp32_model_size = get_model_size(fp32_model_path, \"pytorch\")\n",
    "int8_model_size = get_model_size(int8_ptq_model_path, \"openvino\")\n",
    "print(f\"FP32 model size: {fp32_model_size:.2f} MB\")\n",
    "print(f\"INT8 model size: {int8_model_size:.2f} MB\")\n",
    "print(f\"INT8 size decrease: {fp32_model_size / int8_model_size:.2f}x\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f8697a2-0a51-427f-8245-cda56bb8cf18",
   "metadata": {},
   "source": [
    "### Benchmarks\n",
    "\n",
    "Compare the inference speed of the quantized OpenVINO model with that of the original PyTorch model.\n",
    "\n",
    "This benchmark provides an estimate of performance, but keep in mind that other programs running on the computer, as well as power management settings, can affect performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8806da79-0b3b-403e-a40c-61db6a0f482d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-12-04T14:44:37.575841Z",
     "iopub.status.busy": "2022-12-04T14:44:37.575681Z",
     "iopub.status.idle": "2022-12-04T14:44:53.148782Z",
     "shell.execute_reply": "2022-12-04T14:44:53.147790Z",
     "shell.execute_reply.started": "2022-12-04T14:44:37.575830Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz\n",
      "Latency of original FP32 model: 113.28 ms\n",
      "Latency of quantized model: 41.29 ms\n",
      "Speedup: 2.74x\n"
     ]
    }
   ],
   "source": [
    "def benchmark(qa_pipeline, dataset, num_items=100):\n",
    "    \"\"\"\n",
    "    Benchmark PyTorch or OpenVINO model. This function does inference on `num_items`\n",
    "    dataset items and returns the median latency in milliseconds\n",
    "    \"\"\"\n",
    "    latencies = []\n",
    "    for i, item in enumerate(dataset.select(range(num_items))):\n",
    "        start_time = time.perf_counter()\n",
    "        _ = qa_pipeline({\"question\": item[\"question\"], \"context\": item[\"context\"]})\n",
    "        end_time = time.perf_counter()\n",
    "        latencies.append(end_time - start_time)\n",
    "\n",
    "    return np.median(latencies) * 1000\n",
    "\n",
    "\n",
    "original_latency = benchmark(hf_qa_pipeline, validation_dataset)\n",
    "quantized_latency = benchmark(ov_qa_pipeline_ptq, validation_dataset)\n",
    "cpu_device_name = Core().get_property(\"CPU\", \"FULL_DEVICE_NAME\")\n",
    "\n",
    "print(cpu_device_name)\n",
    "print(f\"Latency of original FP32 model: {original_latency:.2f} ms\")\n",
    "print(f\"Latency of quantized model: {quantized_latency:.2f} ms\")\n",
    "print(f\"Speedup: {(original_latency/quantized_latency):.2f}x\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
